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•  Introduction 

•  Cognitive  Systems  Engineering 

•  Systems  Engineering  Analysis 

•  Visualization  Interface  Concept  Development 

•  Concept  Evaluation 

•  Conclusions  and  the  Way  Ahead 


Operational  Effects  Assessment 
Visualization  Tool  (OEAVT) 


Objective 


Decision-quality  support  to 
assessment  team  in  defining 
critical  indicators,  managing 
assessment  data,  determining 
operational  effectiveness, 
visualizing/understanding 
complexity  and  uncertainty. 


Approach 


•  Analysis  of  assessment  domain 

•  ID  assessment  functions 

•  ID  requirements 

•  Design/Refine-Prototype 

•  Build 

•  Operational  test 


Benefits  to  Warfighter 


•  Decision-quality  knowledge 
to  the  commander 

•  Continuous  operational 
assessment  tied  to 
objectives 

•  Sensemaking  of  battlespace 
effects 

•  Uncertainty  management 

•  nth-order  causal 
understanding 

•  Faster  decision  times 


Technologies 


•  Information  Visualization 

•  Knowledge  Management 

•  Intelligent  Interface  Agents 

•  Advanced  Search/Data  Mining 


Evidence  Accrual 


Operational  Assessment 


Prediction  &  Forecasting 


Indicator  Mgmt 
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Effects- based  Assessment:  Operational 

Issues 


•  All  operations  have  complex 
effects. 
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-  Desirable  and  undesirable 
effects 


-  1^^,  2^^  and  order  effects 


-  Time  delays  introduce 
uncertainty  and  risk 

Understanding  the  implications 
of  operations  is  a 
multidimensional  problem. 

-  Effects  can  be  strategic, 
operational,  physical, 
psychological,  ... 


-  System  of  systems  is  a  major 
conceptualization  &  modeling 
challenge 
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ffects-based  Assessment:  Human 

Issues 
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•  Data  management 


-  What  to  measure,  when  to 
measure,  how  often  to  measure? 

•  Dynamic  assessment 

-  Finding  appropriate  indicators 
and  measures 

-  Integrating  results  in  real  time:  A 
data  aggregation  problem,  an 
interpretation  problem. 

•  Plan  troubleshooting 


-  How  does  one  decide  when  to 
"stay  the  course"  or  to 
recommend  changes  the 
strategic  plan? 

-  How  to  evaluate  the  efficacy  of 
potential  changes? 


Campaign  Assessment 


Combat  Assessment 


Cognitive  System  Engineering 
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EBA  Domain  analysis  and 
cognitive  system 
engineering 


System  engineering 
analysis  and  requirements 


Visualization  concept 
development 


Develop  prototype 


Concept  Mapping:  Basic  Information  From  Science  to  Solutions' 


•  Five  "views' 


Q  Work  management 


•  Some  overlap 


varies-over/ 

tailor 


Cognition 


n  Products 


G  Collaboration 


Automation 


|_  supports/  ^1 
SLippoitecl-hy 


-  == 

Concept  Mapping 
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Identified  the  following  cognitive 

work  for  EBA: 

Pre-execution 

Execution 

Post-execution 

•  Assessment  planning 

•  Accrue  evidence 

•  Inter-division 

•  Determine  adversary 

•  Analyze  ops  results 

feedback 

capabilities  &  likely 

•  BDA 

COA 

•  Develop  JAOP 

•  Develop  STTM 

•  Execution  tracking 

•  Functional  damage 
assessment 

•  OAT  mgmt  of  EBA 

•  Integrate  mission 

•  Predict  ops 

assessment 

effectiveness 

•  Mission  assessment 

•  PDA 

•  Target  system 
assessment 
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Decision  Requirements  Tables 
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•  Identify  and  characterize  assessment 
decisions  that  will  drive  visualization 
requirements 

-  Task 

•  Critical  cues 

•  Critical  decisions 

•  Common  errors 

•  Actions 

•  Tools  used 

•  Collaboration  and  communication 

•  Data  used 

•  Requirements 
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DRT  Example 
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•  Recognize  actionable  changes  in  ongoing  air  ops 

•  Assess  feasibility  of  plan  changes 


Critical  Cues 

Critical 

Decisions 

Actions 

Common 

Errors 

Tools  Used 

Communicates 

with 

Data  Used 

Requirements 

^Change  in 
Dathways 

^Change  in 
ift/eather 

*Etc. 

^Determine 
when  changes 
have  been 
made  in 
ongoing  air 
ops 

^Monitor  activity 
trends  in  areas 
indicated  by 
critical  cues 

^Monitor 

indicators 

against 

predictions  and 
time 

*TBMCS 

*ISRD 

*Plans  team 

*MISREPS 

*INTSUMS 

*DISUMS 

*Combat 

assessment 

*The  system 
shall  allow  and 
aid  in 

recognizing 
actionable 
changes  in 
ongoing  air 
ops 

•WOE 

•  Time 

•Guidance 

•Resource 

Drofiles 

^Determine  if 
resources 
available  for 
corrective 
actions 

*  Determine 
what  to 
change  & 
amount  of 
change 

^Determine 
WOE  for  each 
objective 

^Determine 

priorities 

*Make 

recommendation 

^Predict 
intended  and 
unintended 
effects  of 
changes 

*Infer  2nd, 

S^d  order  effects 
associated  with 
potential 
changes 

*IWPC 

*TBMCS 

*ISRD 

*Plans  team 

*Combat  Ops 

*TPFID 

*ATO 

^Guidance 

*The  system 
shall 

determine  if 

resources  are 
available  for 
corrective 
actions 

*The  system 
shall 

determine 
amount  of 
change  to 
plan,  and  what 
to  change 

Td 


CORE  Systems  Analysis 
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•  CORE:  Commercially  available  CASE  tool 

-  Allows  management  of  the  whole  project 

-  A  wide  range  of  information 

-  DES  supports  tradeoff  &  what-if  analysis 

-  DoDAF  compatibility 

•  What  information  we  put  into  it 

-  All  concept  map  and  CDA  information 

-  Other  information  specified  by  SMEs 

•  Vetted  with  SME  input 

•  108  functions 
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CORE  Diagram  Examples 
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Execution 
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Visualization  Interface  Concept  Development 
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Scenario 

Element 


Operational  execution 


Operational 

Requirement 


Indicator 

Management 


Evidence 

Accrual 


Prediction  and 
Forecasting 


i 


Capability/Support 
Requirement  Data 


Data  mgmt  Cascading  ^ 
effects  Causal 

Temporal  Unk 

mining  assessment  analysis 


Operational  Reporting 

Assessment 


Plan 

troubleshooting 


-i 


Uncertainty 

visualization 


“Live” 

briefing 

capability 


Functional 

Requiremem 
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OEAVT 

TMia 

iria  EM  VteiiM  WAi 

lb*  tup 

js. 


New  Evidence:  1 0 


^A;S.  Gain  and  maintain  air  and  space 
superiority  to  ensure  freedom  of  action 
for  Coalition  forces 

SI:  ^r superiority  is  achieved  over 
designated  ZOCs  (ZOCs  3.  4,2,  and 
1  in  that  order)  and  eventually  over 
the  entire  air  space  of  CN  by  D  +  24 

B  AS1 .  Protect  Coalition  resources 
from  air  and  missile  ^ack 
MOE:  No  significant  loss  of 
P  DAL  assets:  No  prohibitive 
losses  of  friendly  assds 

y  ASIA  Defeat  enemy  fixed  and 
rotary-wing  attacks  into  NV 

MOP:  No  sucessful  air 
attacks  on  PDALassds:no 
loss  of  HVAA  CN  offensive 
air  capability  is  minimally 
effective  NLT  D-h7 
AS  1 B .  Defeat  enemy  S  SM  attacks 

MOP:  P DAL  assets  are 
effectively  defended  from 
SSM  attacks  NLT  D-h1 


[+]  V  AS2.  Disrupt/Degr^de  CN  IADS 

Quons  QtJTM 

MOE:  CN  defensive  air 

1 

capability  is  minimally  effective 

1  t 

NLT  D-hID;  Coalition  air  can 

fua  1 

provide  CAS  in  the  PMF  without 

prohibitive  losses  on  D-h1 

0«n«ral  w 

+]  CL.  Support  CFLCC's  operations  to 

culminate  Calif  on's  ground  offensive  NLT 
D-h14  and  then  render  CN  unable  to 
defend  effectively 


File  Edri  V?ew  Help 

i*  *  r  ^ 

fxm  “1^ 


I  TM* 

I  On3 

gmgwnfiiiTwi 

$  ■  Ttrnh 

^  W 

V  Tniii  porti-ftH 


m  »m4  int  SVOT  fcEKH'  li  W 


Concept  Evaluation 


From  Science  to  Sofutions^** 


•  Purpose:  Demonstrate  visualization 
concepts  for  an  integrated  ops  assessment 
system 

-  Feedback  on  concept 

-  Feedback  on  implementation 
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Conclusions 
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This  methodology  enables  a  meaningful  integration 
of  cognitive  systems  analysis  with  accepted  system 
engineering  and  technology  development  practice. 


The  cognitive  and  perceptual  work  involved  in  EBA 
can  be  captured  by  a  limited,  manageable  number  of 
hierarchically  structured  functions. 


Visualization  technologies  must  be  both  broad  and 
deep  for  success  in  an  EBA  domain. 
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The  Way  Ahead: 
Potential  Long-term  Direction 
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Development  Focus 

Potential  Technology  Solutions 

PMESII  visualization 

-  Hierarchical  task  network  with  recursive  task 

blocks 

-  3-D  rendering 

-  Fisheye  view  on  demand 

Intelligent  queries 

-  Queries  for  spatial,  temporal  and  probabilistic 

content 

-  Intelligent,  automated  data  acquisition 

Causal  link  analysis 

•  between  actions  and  effects 

•cause  -  effect  latencies;  latencies  in 
observing  effects 

•  temporal  effects 

-  Influence  nets 

-  Colored  Petri  Nets 

-  Hybrid  dynamical  systems 

-  Temporal  causal  graphs 

-  Causal  graphs  &  event  calculus 

-  Timed  failure  propagation  graphs 

N’^'^-order  effects 

-Temporal  causal  graphs 
-  Causal  graphs  &  event  calculus 
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Questions 
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